Enforce working examples in docstrings and documentation #1588
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
strawgate
changed the title
strawgate
changed the title
strawgate
changed the title
strawgate
changed the title
strawgate
force-pushed
the
add-pytest-examples
branch
from
1a46a09 to
5fe8ece
Compare
strawgate
force-pushed
the
add-pytest-examples
branch
from
5fe8ece to
a23907e
Compare
|
Hey sorry, should have left this in draft, I'll pick this back up and get it ready to be merged |
|
@alexmojaki thanks for the review, I think this is looking better now. |
|
@alexmojaki i did an audit of the 20 uses of hl_lines in the docs, resolved all issues I could find (whether caused by this PR or currently mishighlighted on main), and manually verified the correct lines are highlighted in the served docs with this PR |
There was a problem hiding this comment.
Pull request overview
This PR introduces comprehensive testing infrastructure to ensure that code examples in documentation and docstrings are properly formatted and runnable. It uses pytest-examples to validate ~300 examples across the codebase, with ~140 examples appropriately skipped when they require external connections, are incomplete, or would block execution.
Changes:
- Added
pytest-examplesandpytest-timeoutas dev dependencies to validate documentation examples - Created
tests/test_docs.pywith formatting and execution tests for code examples - Updated documentation examples throughout to be properly formatted, with correct import ordering and skip tags
- Added MkDocs plugin code to strip custom test attributes from rendered documentation
Reviewed changes
Copilot reviewed 71 out of 73 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_docs.py | New test file implementing example validation with pytest-examples |
| pyproject.toml | Added pytest-examples>=0.0.18 and pytest-timeout>=2.4.0 to dev dependencies |
| uv.lock | Updated dependency lockfile with new packages and version bumps for dspy and gepa |
| docs/plugins/main.py | Added function to strip custom test attributes from code blocks in rendered docs |
| logfire/integrations/pydantic.py | Fixed import ordering in docstring example |
| logfire/_internal/main.py | Added skip tags to examples requiring external connections and fixed import ordering |
| docs/**/*.md | Extensive formatting fixes including import ordering, skip tags, and consistent quote usage |
| README.md | Updated examples with proper formatting and skip tags |
| Makefile | Added test-update-examples target for updating documentation examples |
| CONTRIBUTING.md | Updated contributor instructions to include new example update command |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
2 participants
Require formatted and runnable examples in docstrings and documentation, and bring the documentation and code up to compliance with these new requirements. Closes #1587
Uses
pytest-examplesto ensure that examples in docstrings and documentation are properly formatted and runnable with assertions on anyprintstatements in the example. Addspytest-timeoutto the dev dependencies which is used to prevent example tests from running forever if they hit blocking code.There are ~300 newly passing tests and ~140 skipped tests (examples that are not checked for formatting or if they are runnable).